Collecting Execution Statistics of Scientific Workflow on Hadoop YARN

The development of computers, sensors and other technical instruments changed scientific research. Current scientific experiments often contain a whole range of computational activities, e.g: capturing and saving huge amounts of data in databases or flat files, analysis of data with software and data visualization [1]. The complexity of these data intensive experiments led to new challenges. Developing and maintaining the physical infrastructure to store the data and execute the analysis pipelines is expensive [2]. Data provenance is also an important aspect scientists need to ensure the origin of the results and the underlying data. They need to share the knowledge and resulting information with the community in order that other scientists can repeat and review the experiments. Scientific workflows provide a means for representing and managing such analysis pipelines. An activity of the analysis pipeline is encapsulated in a workflow step (task). A scientific workflow is a directed, acyclic graph (DAG), in which individual tasks are represented as nodes. Edges represent data dependencies between tasks. One task can be local software or a remote (web) service call which transforms input data to output data. Task execution is constrained by data dependencies [3]. An abstract workflow is a high-level workflow description. Scientists model an abstract workflow by specifying a set of individual tasks and the data dependencies between them [4]. Figure 2 shows an abstract workflow. Scientific workflow management systems (SWfMS) are used to manage and execute scientific workflows. In addition, they often support scientists to record provenance information and statistics about the execution of the workflows. Provenance data is supposed to trace the flow of data through the workflow tasks to reason about the results. Statistics about the execution of workflows and tasks are useful not only for provenance questions: Precise information about historical executions of a workflow can help the SWfMS to execute the workflow faster or to present progress and time-remaining estimation [5].